System and method for laser imaging and ablation of cancer cells using fluorescence

ABSTRACT

A fluorescence imaging device detects fluorescence in parts of the visible and invisible spectrum, and projects the fluorescence image directly on the human body, as well as on a monitor, with improved sensitivity, video frame rate and depth of focus, and enhanced capabilities of detecting distribution and properties of multiple fluorophores. Direct projection of three-dimensional visible representations of florescence on three-dimensional body areas advantageously permits view of it during surgical procedures, including during cancer removal, reconstructive surgery and wound care, etc. A NIR laser and a human visible laser (HVL) are aligned coaxially and scanned over the operating field of view. When the NIR laser passes over the area where the florescent dye is present, it energizes the dye which emits at a shifted NIR frequency detected by a photo diode. The HVL is turned on when emission is detected, providing visual indication of those positions.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority on U.S. Provisional Application Ser.No. 61/733,535 filed on Dec. 5, 2012, and claims priority on U.S.Provisional Application Ser. No. 61/830,225, filed on Jun. 3, 2013, withthe disclosures of each incorporated herein by reference.

FIELD OF INVENTION

This invention is related to the fields of fluorescent medical imagingand the laser ablation treatment of cancer.

BACKGROUND OF INVENTION

Fluorescence is a phenomenon of light emission by a substance that haspreviously absorbed light of a different wavelength. In most cases, theemitted light has a longer wavelength, and therefore lower energy, thanthe absorbed light. However, when the absorbed electromagnetic radiationis intense, it is possible for one atom or molecule to absorb twophotons; this two-photon absorption can lead to emission of radiationhaving a shorter wavelength than the absorbed radiation. Fluorescentlight can be easily separated from reflected excitation light, thusproviding excellent selectivity in applications where fluorescent lightmay carry some useful information about the substances and structureswhich emitted it.

This property is particularly important in various medical imagingapplications, where fluorescent light may be emitted by fluorescentdyes, also known as fluorophores, with affinity to certain biologicalmaterials such as blood, or dyes conjugated to biological markers withspecific affinity to certain tissues, proteins or DNA segments, and canbe a reliable proxy for imaging internal body structures, such as bloodvessels, lymph nodes, etc., as well as finding signs of disease, such asnecrosis or cancer.

Usually, fluorescent biological markers are introduced externally,specifically with a purpose of binding to and imaging specific organsand tissues. In some cases, they are naturally-occurring, which is knownas biological auto-fluorescence.

Most fluorescent substances, whether biological or not, have specificabsorption and emission spectra, with peaks at certain wavelength.Sometimes, more than one peak may be present in either absorption oremission spectrum, or both. In any case, any fluorescent imaging systemmust provide excitation light at one wavelength and detect the emissionlight at different wavelength. Since the optical efficiency offluorescence is usually quite low, emission light is usually much weakerthan excitation light. Hence, optical filters which accept emissionlight and block excitation light are also usually present in afluorescent imaging system.

Of particular interest are the fluorescent dyes which both absorb andemit light in the Near Infrared (NIR) part of the spectrum,approximately from 700 to 1000 nm wavelength. Within this band, humantissues are particularly transparent, so the fluorescent dyes may beseen at most depths and images may be of particular clarity.

Fluorescent medical imaging systems are known in prior art, includingthose designed for detection of NIR fluorescent dyes.

Usually, fluorescent images are combined with conventional,reflected-light images and presented to a medical practitioner on acommon monitor, so the distribution of the fluorescent die can bevisible in its surroundings. Since the NIR fluorescent image is outsideof the human visible light range, it is usually mapped to a humanvisible color and displayed on a monitor superimposed on top of thecaptured color image of the biological object of interest. The systemcan display either still or moving images. A medical practitioner canuse such a system to detect, for example, cancer cells during surgery,detect perfusion during reconstructive surgery, and detect the locationof lymph nodes. During open surgery, wherein the surgeon is directlyviewing the field of surgery, utilization of a monitor isdisadvantageous in the surgeon must glance away from the surgical siteto view the image of the fluorescence. Upon returning to view thesurgical area, the surgeon must estimate the position of the florescencebased upon his memory of the display on the monitor. Alternative, thesurgeon can perform the required work while directly viewing the monitoras opposed to directly viewing the surgical area. This approach isdisadvantaged in that it is cumbersome to operate without directlyviewing the surgical site. Further, when viewing the monitor the surgeonlosses all three dimensional information that is normally obtained whendirectly viewing the surgical area.

While being valuable surgical and diagnostic tools, known fluorescentcameras suffer from a number of limitations, mostly stemming from verylow signal levels produced by fluorescent dyes. Those limitations areinsufficient sensitivity, low video frame rates or long integration timenecessary for taking a still image, as well as a limited depth of focus,especially if a large objective lens is used to alleviate sensitivityproblems.

There are many known fluorescent dyes and or molecules that are used inthe medical field, also referred to florescent probes or fluorescentmarkers. (see,www.piercenet.com/browse.cfm?fldID=4DD9D52E-5056-8A76-4E6E-E217FAD0D86B,the disclosures of which are hereby incorporated by reference).

Furthermore, in the following article, near-infrared fluorescencenanoparticle-base probes for use in imaging of cancer are described andis hereby incorporated by reference: He, X., Wang, K. and Cheng, Z.(2010), “In vivo near-infrared fluorescence imaging of cancer withnanoparticle-based probes,” WIRES Nanomed Nanobiotechnol, 2: 349-366.doi: 10.1002/wnan.85.

OBJECTS OF THE INVENTION

It is an object of this invention to visualize fluorescence which isinvisible to the human eye, either because it is outside of visiblewavelength range, or because it is too weak to be seen by a naked eye,by re-projection directly onto human tissue on a surgical or diagnosticsite and thus free the medical practitioner from shifting his sight frompatient to monitor and back

It is another object of this invention to alleviate the deficiencies ofexisting fluorescent cameras and increase the sensitivity, the videoframe rates and the depth of focus.

It is yet another object of this invention to enable a fluorescentcamera to detect the presence of multiple fluorophores with distinctspectra in human tissues and adequately convey the information abouttheir presence and distribution to the medical practitioner.

It is yet another object of this invention to detect temporal changes influorophore distribution and convey this information to the medicalpractitioner.

It is yet another object of this invention to enable detection offluorescence life time of a fluorophore, which might convey additionalclinically-relevant information about fluorophore distribution andinteraction with surrounding tissues. It may also help to distinguishbetween fluorophores with the same spectra but different life time.

It is also an object of this invention to extract information about thedepth of the fluorophore deposit in the human body, thus enabling3-dimensional fluorescence imaging.

And it is also an object of this invention to improve the performance offluorescence imaging in endoscopic applications.

Further objects and advantages of the invention will become apparentfrom the following description and claims, and from the accompanyingdrawings.

SUMMARY OF THE INVENTION

In this application a fluorescence imaging device is described which iscapable of detecting fluorescence in visible and invisible parts of thespectrum and projecting the fluorescence image directly on the humanbody, as well as on the monitor, with improved sensitivity, video framerate and depth of focus and enhanced capabilities of detectingdistribution and properties of multiple fluorophores.

Projecting the visible representation of the florescence directly on thehuman body has the significant advantage of allowing the surgeon to viewthe florescence directly on the patient while performing the surgery.Since the parts of the body being operated on are three dimensional, theviewing by the surgeon of the projected visible image thereon istherefore inherently three dimensional, providing an advantage to thesurgeon.

An illustrative example where the present invention would be useful isopen surgery for cancer removal. It is known that injecting a patientwith fluorescent dyes conjugated with specific biological markers willcause the dyes to accumulate in cancer cells. With the presentinvention, during open surgery the surgeon can simply aim the device atthe surgical area and all of the cancer cells will appear to be visuallyglowing due to the selective projection of the visible laser on thesurgical area. In this manner the surgeon can make certain to onlyremove the cancerous materials, and can insure that all the cancerouscells are appropriately removed.

A further illustrative field is the area of reconstructive surgery andwound care. In these cases insuring that there is appropriate blood flowinto parts of the body is critical. In this instance, during thereconstructive surgery the fluorescent dyes can be injected into thepatients' blood stream and the device used to show the blood flow to thenewly constructed area. By projecting directly onto the reconstructedarea an image of the blood flow, the surgeon can insure in real timethat the flow is appropriate to provide appropriate healing.

In one embodiment, the system includes a NIR laser for energizing theflorescent dye, for example Indocyanine green (ICG). Also included is ahuman visible laser (i.e., a laser emitting light at a wavelength thatis visible to the human eye) for displaying the areas where theflorescence is detected. Both the NIR and the visible laser are alignedco axially and scanned over the operating field of view. When the NIRlaser passes over the area where the florescent dye is present, the dyeemits at a shifted NIR frequency which is detected by a photo diode.Based upon the position of the NIR laser when the emission is detected,the position of the florescent dye is identified. The human visiblelaser is then turned on at positions corresponding to the position ofthe florescent dye, thereby providing a visual indication of theposition of the florescent dye.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1—this block diagram describes a fluorescence imaging devicecapable of re-projecting fluorescence image directly on the human body.

FIG. 2—this block diagram describes a fluorescence imaging devicecapable of re-projecting fluorescence image directly on the human bodyand on a monitor.

FIG. 3—this block diagram describes a fluorescence imaging devicecapable of re-projecting fluorescence image directly on the human body,overlap the fluorescent image on the visible image from a CCD camera anddisplay the combined image on a monitor.

FIG. 4—this block diagram describes a fluorescence imaging devicecapable of re-projecting fluorescence image directly on the human bodyusing full RGB colors, overlap the fluorescent image on the visibleimage from a full-color laser scanning camera and display the combinedimage on a monitor.

FIG. 5—this block diagram describes a fluorescence imaging devicecapable of re-projecting fluorescence image directly on the human bodyusing full RGB colors, overlap the fluorescent image on the visibleimage from a full-color laser scanning camera and display the combinedimage on a monitor. This device has an additional ablation laser,capable of controlled delivery of laser light to select regions of thehuman body to facilitate removal of tissue, which is designated based onthe acquired fluorescence image.

FIG. 6—this block diagram describes a fluorescence imaging devicecapable of re-projecting fluorescence image directly on the human body,overlap the fluorescent image on the visible image from a CCD camera anddisplay the combined image on a monitor. This device uses an imaging,rather than a laser scanning, projector, such as a Digital LightProcessor (DLP) projector.

FIG. 7—this drawing shows a simplified layout of the device of FIG. 3.

FIG. 8—this drawing shows the difference between the imaging andnon-imaging light collection and the advantage of the latter for afluorescence imaging device.

FIG. 9—this drawing shows a simplified layout of an endoscopicfluorescence imaging device.

FIG. 10—these graphs show the difference between temporal responses ofshort and long fluorescent life fluorophores while excited by a laserscanning beam.

FIG. 11—this drawing shows a simplified optical Field-of-View (FOV) of afluorescence imaging device.

FIG. 12—these timing diagrams illustrate the process of simultaneouslyacquiring a fluorescent image and projecting with a fluorescence imagingdevice.

FIG. 13—this drawing shows a method of optically combining the FOV oflaser scanner and a CCD camera of a fluorescence imaging device.

FIG. 14—this drawing shows an alternative method of optically combiningthe FOV of laser scanner and a CCD camera of a fluorescence imagingdevice.

FIG. 15—this drawing shows yet another method of optically combining theFOV of laser scanner and a CCD camera of a fluorescence imaging device.

FIG. 16—this drawing shows a method of optically combining the FOV of animaging projector, such as a DLP projector, and a CCD camera of afluorescence imaging device.

FIG. 17—this drawing shows a fluorescence imaging device with ahead-mount sensor for correcting the re-projected image.

FIG. 18—this diagram illustrates the visual enhancement of re-projectedimage through synchronized blinking.

FIG. 19—these timing diagrams illustrate time-resolved fluorescencedetection.

FIG. 20—these timing diagrams illustrate and alternative method oftime-resolved fluorescence detection using a single-photon counter.

FIG. 21—this drawing illustrates the optical collection area trade-offbetween imaging and non-imaging detectors.

FIG. 22—this drawing shows multiple detectors with independentcollection areas covering the same FOV.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a block diagram of a fluorescence imaging device capable ofre-projecting fluorescence image directly on the human body, forexample, during a surgery. A florescent dye, such as HER2Sense orRediJect 2-DG 750, which is available from Perkin Elmer in Waltham,Mass. (see, www.perkinelmer.com/catalog/category/id/Targeted), isdelivered to the surgical area, via sub-cutaneous or intra-venousinjection and accumulates in the objects of interest, for example cancercells 11 of the patient.

A near IR laser 1 is provided at an IR frequency that is suitable forexciting the florescent dye in the cancer cells 11 of the surgical area.The near IR laser 1 passes through an alignment mechanism 3 which coaxially aligns the near IR laser 1 with a visible laser 2. As a specificillustrative example, the near IR laser 1 could be a semiconductor laserdiode which emits at 780 nm wavelength and the visible laser 2 can be ared laser that emits at a 640 nm wavelength. The co-axially alignedlasers beams are then delivered to a scanning mechanism 5 which movesthem in a raster pattern along a field of view 9 aimed upon the surgicalarea 7.

When the near IR laser passes over the cancer cells 11 the florescentdye contained therein is energized and emits light in a band roughlycentered around an 820 nm wavelength. The emitted light travels alongdetection path 10 to a filter lens 6. The filter lens 6 has opticalcharacteristics that allow the 820 nm wavelength light traveling alongdetection path 10 to pass through the filter lens 6, and is focused bythe filter lens 6 onto photodetector 4. The filter lens 6 also hasoptical characteristics that block the 780 nm near IR laser light frompassing through to the photodetector 4. The photodetector 4 converts the820 nm light emitting from the cancer cells 11 into an analog signalwhich is then provided processing unit 12.

In one embodiment, called the real-time mode, the processing unit 12drives in real time a laser drive 13, which in turn turns on and off thevisible laser 2 so that the visible laser 2 represents the amount of the820 nm wavelength light falling upon the photodetector 4. In thismanner, the visible laser 2 is transmitted onto the surgical area 7thereby illuminating with visible light the locations where thefluorescent dye is present.

In another embodiment, called an image capture mode, the processing unit12 stores a time sequence output of the photodetector 4 into a memorylocation. In this manner, an entire image frame representing the 820 nmfluorescence in the field of view 9 is stored in the memory location.Image processing can be performed on the memory location to enhance andaugment the captured image. The processing unit 12 then outputs a timesequence output to the laser drive 13 such that the visible laseroutputs the entire frame stored in the memory location. In this matter,the frame captured in the memory location is transmitted onto thesurgical area thereby illuminating the visible light at the locationswhere the fluorescent dye is present. In this image capture mode, theoutput frame from the visible laser 2 is delayed in time from the imageframe stored by the processing unit 12. The device of FIG. 1 can becontained in handheld devices or can be mounted on a stand. In thehandheld device, provided the frame rate is adequately fast, forexample, 60-100 frames per second, this delay will not result innoticeable jitter.

A further embodiment of the device of FIG. 1 is shown in FIG. 2. Theelements 1-13 function in the image capture mode as described inreference to FIG. 1. Further, the processing unit 12 communicates theframe image stored in the memory representative of the 820 nmfluorescence in the field of view 9 through communications 14 circuitryto either a monitor 15 or storage 16 device, or to both the monitor 15and the storage 16 device. The monitor 15 displays the frame image ofthe fluorescence in the field of view 9. Further, the storage 16 device,such as, for example, a hard drive, solid state memory device orcomputer, can store the frame image and later recall the image forviewing on the monitor 15 or for archiving the frame images.

The user of the FIG. 2 embodiment will have the ability to view a visualimage of the 820 nm fluorescence in the field of view 9 generated by thevisible laser 2 and scanned by the scanning mechanism 5 directly on thesurgical area 7. Surgical area 7 often has a three-dimensional geometry,and the visible image is displayed on all portions of thethree-dimensional geometry of the surgical area facing the direction ofthe scanning mechanism. Further, the user can view the visual image ofthe 820 nm fluorescence in the field of view 9 directly on the monitor15. The display on the monitor can be digitally amplified or reduced tofit the user's needs.

Another embodiment of the present invention is shown on FIG. 3. All theelements are the same as FIG. 2; however, a CCD Camera 20 has beenelectrically connected to the processing unit 12. While the invention ofFIG. 2 is operating, the CCD camera 20 takes a color image of thesurgical field 7. Accordingly, the CCD camera captures an image of thesurgical field 7 while the visible laser 2 is projecting a visiblerepresentation of where the florescent material is. In this manner, theCCD camera 20 captures and communicates to the processing unit 12 animage representation of both the color image of the surgical field 7together with the projected representation of the fluorescence. Suchcombined image can then be displayed on the monitor. In order to displayjust the visible image, without the fluorescence the near IR laser canbe temporarily turned off thereby stopping the fluorescing and therebythe CCD camera images just the visible image.

The projected visible image captured by camera can also be analyzed bythe processing unit in order to provide the most accurate alignment ofthe projected image on curved surfaces of the body with the capturedfluorescent image.

A further embodiment of the present invention is shown in FIG. 4 whereina color representation of the surgical area 7 is captured together withthe fluorescent representation of the surgical area 7.

Within a surgical field a surgical area is treated with a florescentdye, the florescent dye accumulates in, for example, the cancer cells 11of the patient. The florescent dye can be injected into the blood streamof the patient or can be locally injected near the suspect cancer cells11.

A near IR laser 1 is provided at an IR frequency that is suitable forexciting the florescent dye in the cancer cells 11 of the surgical area.The near IR laser 1 passes through an alignment mechanism 3 which coaxially aligns the near IR laser 1 with a green laser 2A, a blue laser2B and a red laser 2C. As an specific illustrative example, the near IRlaser 1 could be a semiconductor laser diode which emits at 780 nmwavelength, the visible red laser 2C can be a 640 nm semiconductor redlaser, the visible blue laser 2B can be a 440 nm semiconductor bluelaser, and the visible green laser 2A can be a can be a laser emittingin the a 510 to 540 nm range. The co-axially aligned lasers are thenprovided to a scanning mechanism 5 which move the coaxially alignedlaser beams in a raster pattern along a field of view 9 aimed upon thesurgical area 7.

When the near IR laser passes over the cancer cells 11 in the surgicalarea 7, the florescent dye contained therein is energized and emitslight in a band roughly centered around an 820 nm wavelength. Theemitted 820 nm wavelength light travels along detection path 10 to alens 6A. The lens 6A has optical characteristics to focus the 820 nmwavelength light traveling along detection path 10 onto photodetector IR4D. A 820 nm pass filter 17D is provided which allows the 820 nmwavelength to pass while rejecting visible light reflected off thesurgical area 7 from the red laser 2C, the green laser 2A and the bluelaser 2B laser as well as rejecting the Near IR light from reflectingoff the surgical area from the near IR laser 1. The 820 nm pass filter17D is positioned between the lens 6A and the Photodetector IR 4D. Inthis manner the photodetector IR 4D receives only the 820 nm fluorescentemission from the surgical area 7 and converts the 820 nm light emittingwithin the surgical area 7 into an analog signal which is then providedprocessing unit 12. The processing unit 12 converts the analog signalfrom photodetector IR 4d into a digital signal which is stored on aframe by frame basis in 820 nm frame memory 18 d

When the green laser 2A, blue laser 2B and Red laser 2C passes over thesurgical area 7 within the field of view 9, the visible colorcharacteristics of the surgical area 7 are reflected to varying degreesdepending upon the visible color of the surgical area 7. The reflectedlight travels along detection path 10 to a lens 6A. The lens 6A hasoptical characteristics to focus the reflected green laser 2A, bluelaser 2B and red laser 2C light traveling along detection path 10 ontoeach of photodetectors 4A-4C, respectively. A green pass filter 17A, ablue pass filter 17B and a red pass filter 17C, which allows only theirrespective colors of visible light to pass through, are positionedbetween the lens GA and the respective photodetectors 4A-4C. In thismanner each of the respective photodetectors 4A-4C receives only one ofthe three reflected colors, and each photodetector 4A-4C converts therespective light into analog signals which are then provided toprocessing unit 12. The processing unit 12 converts the analog signalfrom the respective photodetectors 4 a-4 c into a digital signal whichis stored on a frame by frame basis in green frame memory 18 a, blueframe memory 18 b and red frame memory 18 c, respectively.

In this manner, an entire frame representing the 820 nm fluorescence inthe field of view 9 together with a color image of the surgical area 7within the field of view 9 is stored within frame memory 18 a-18 d ofthe processing unit 12. To directly illuminate the areas within thesurgical area 7 that emitted the 820 nm light, the 820 nm frame memory18 d is mapped to a selected color for projection onto the surgical area7. For example, if a red color is selected as the display color, theprocessing unit 12 outputs a time sequence of the frame within the 820nm frame memory to the red laser drive 13 c such that the red laser 2c-k is driven to output onto the surgical area the image stored withinthe 820 nm frame memory. Accordingly, the surgeon will see directly onthe surgical area 7 the red laser projection at the locations where the820 nm fluorescence occurred. While in the present embodiment, the redlaser 2C was utilized for projecting the visible image onto the surgicalarea 7, in alternative embodiments, any desired combination of the redlaser 13 c, the blue laser 13 b and the green laser 13A could be used toproject a desired visible color.

In the present embodiment, the image contained in the 820 nm framebuffer can mapped to a visible color and superimposed onto one or moreof the green, blue or red frame memories 18 a-18 c and the resultingmodified frame memories 18 a-18 c are then displayed on monitor 15 andoutput to storage 16. For example, in an embodiment wherein bright greenis selected as the color for displaying on the monitor 15 the image ofthe fluorescence stored in 820 nm frame memory 18 d, then green framememory 18 a is modified based upon the contents of 820 nm frame memory18 d, such that bright green is stored in green frame memory 18 a at thelocations where the 820 nm frame memory 18 d stored indication offlorescence detection.

Accordingly, with the present invention the surgeon has two ways ofviewing fluorescence within the surgical area 7. In the first, thevisible lasers (one or more of the green, blue and red lasers 18 a-18 care projected directly on the surgical site and directly locations whichare fluorescing. Further, the image of the fluorescing is mapped to acolor and display on the monitor 15 together with a color image of thesurgical area 7.

In this embodiment, the color lasers 2A-2C are used to illuminate thesurgical area 7 to capture a color image, and one or more of the colorlasers is used to project the representation of the areas offluorescence. This can be accomplished by time multiplexing the lasers2A-2C. For example, every other frame can be allocated for the captureof the color image and the alternate frames can be allocated todisplaying via the one or more color lasers 2 a-2 c the representationof the fluorescence. The net effect will be a white background with theimage of the florescence superimposed thereon.

There exists a large number of fluorophores which can be utilized withthe present invention. Each fluorophores is activated by particularfrequency of light, and emits a particular frequency of light. It isunderstood that the Near IR laser 1 can be of any frequency sufficientto activate the emissions of the fluorophore, and the 820 nm pass filter17 d and the photodetector IR 4d, can be modified to allow the frequencyof light emitted by the fluorophore to be passed and detected. In thismanner the present invention is applicable for the wide array offluorophores. In a still further embodiment, it is possible to utilizetwo or more fluorophores, having different optical characteristics, atthe same time with the surgical area 7. The device of FIG. 4 can bemodified so that there are additional lasers incorporated for activatingthe fluorophores, and additional pass filters and photodetectors fordetecting the different frequencies of light, emitted by thefluorophores. Controls can be incorporated to select which lasers shouldbe activated based upon the fluorophores utilized in a procedure.Further, an auto select mode can be implemented where each laser forexciting the fluorophores is momentarily turned on, and only ifexcitation light is received from the corresponding fluorophores is thatchannel used in the steady state operation of the device.

FIG. 5 is the same as FIG. 4 with the addition of an ablation laser 21.In an embodiment wherein the florescent dye is introduced to bind tocancer cells 11, in addition to causing the visible light to illuminatecancer cells 11, an ablation laser 21 can be controlled so that it turnson only when the lasers are aimed by the scanning mechanism at thecancer cells.

In an alternative embodiment, the scanning mechanism can particularly bea pointing mirror (as opposed to a resonance mirror). In this manner,the ablation laser 21 can be aimed at the desired cancer location for aprolonged period of time to enable sufficient heating to destroy thecells.

Early success with laser ablation on various types of cancer patientshas been achieved at the Mayo Clinic (see e.g.,http://www.mayoclinic.org/news2010-jax/6006.html). The device of FIG. 5can be used to more particularly control the aiming of the laser so thatit falls only on the desired locations.

FIG. 6 is an embodiment wherein a projector (which can be of any type,for example, laser projector, DLP projector, LCD projector, isconfigured solely for projecting visible light of one or more colors. AnIR light source, at a frequency sufficient to cause a fluorophore toemit a different frequency of light, is aimed at the surgical site. TheIR light source can either flood the surgical site or can be a scannedlight source. A camera is configured to detect a wide frequency range oflight, including the visible spectrum and the frequency emitted by thefluorophore. The captured image is stored in a processing unit whereinit is then displayed on a monitor and also could be stored in storagefor record keeping. Further the portion of the captured imagecorresponding to the frequency emitted by the fluorophore, in this case820 nm, is provided to the projector which in turn projects the imageonto the surgical area. Accordingly, a surgeon can see the florescenceby either viewing the monitor or directly looking at the surgical area.

Embodiments presented on FIG. 1 . . . 6 are further illustrated with asimplified layout of FIG. 7. Light collection system 103 insures thatthe light emitted by fluorophore particles reaches the light detectors108. Filters 4 are chosen to correspond to the emission bandwidth offluorophores 105. Detectors 108 convert light into electrical signalswhich are processed in electronic block 109, which forms a 2D imagecorresponding to the distribution of fluorophores in tissue. Said imageis presented on the monitor 110.

Some of the detectors 108 and filters 104 may be configured to receivethe reflected light from excitation lasers 101 or projection lasers 111(of which more below), in addition to fluorescence light. That enablesthe device to act like a camera in IR and/or visible bands. Electronicblock 109 may also perform various image-enhancing processing steps,such as integration over multiple frames, contrast enhancing, etc.

In addition to color mapping, the electronic block 109 is alsoresponsible for brightness mapping of certain levels of light emitted byfluorophores to corresponding levels of power of the projection lasers.Such mapping could be linear, single binary threshold, etc.Additionally, the electronic block 109 may produce other video effectsto emphasize certain features of the image, such as blinking or periodiccolor changes.

It is also possible to modulate the brightness of the illuminationlasers in accordance with the distribution of light collected from thefluorophore. Applying more illumination where fluorescence is weak andless illumination where it is strong would increase the effectivedynamic range of acquired image.

Since the light emitted by fluorophores is usually scant, thecorresponding electrical signals are week and susceptible to noise. Tooptimize image quality, the electronic block may be performingon-the-fly noise estimates and adjust the brightness mappingaccordingly. Additionally, the electronic block may tune the bandwidthof the signal processing tract depending on the minimal feature size inthe observed fluorescent image.

In clinical practice, it is often important to overlap the imagerepresenting fluorescent signature of the tissue with a regular image ofthe same area. To achieve that, an imaging camera 112 can be employed,looking at the same field of view as the scanner. The camera will pickup both the reflected colors of the tissue and the image re-projected bythe projection lasers. Preferably, colors distinct from tissue colorsshould be chosen for re-projection. It is also beneficial to synchronizethe frame rate of the camera with that of the scanner.

Detectors 108 are typically photo-diodes (PD), with appropriateelectronic gain down the signal tract. However, in order to improvesignal-to-nose (SNR) ratio and facilitate detection of very weaksignals, a photo-multiplier tube (PMT) may be used.

Also, to improve fluorescent light collection, a non-imaging lightcollection system can be used, since non-imaging light collectors can besubstantially larger than imaging ones. The difference between them isillustrated on FIG. 8. The imaging collection system 115 has the abilityto collect light from a point A or B on the target into a point A′ or B′on the detector. A non-imaging system 116 can only collect light from apoint on the target into a relatively large area (AA′ or BB′) on thedetector, making it unsuitable for use with pixelated sensors. In ascanning device, however, only temporal resolution of the detectormatters. Refractive, diffractive or reflective non-imaging collectorsmay be used.

The use of very large collectors in conjunction with PMT or otherhigh-sensitivity detectors enables imaging of internal tissues, wherethe scanned raster of the excitation and projection light is deliveredthrough the endo scope 113 (FIG. 9), while the fluorescent light iscollected externally, coming through skin 117. A miniature endoscopiccamera 114 may still be used to produce a combined image withfluorescent features superimposed over regular optical image. Aminiature endoscopic camera in and of itself is typically incapable ofpicking up weak fluorescent light.

Additional advantage of scanning fluorescence detector is its ability toresolve signals in time domain, thus distinguishing between fast- andslow-decaying fluorophores, even if their emission spectra areidentical. FIG. 10 shows an excitation laser beam 118 scanning acrossfluorophore particle 105 and the temporal graphs of excitation light 119and responses of fast (120) and slow (121) fluorophores. To increasetime resolution, the excitation laser itself may be modulated and thetemporal response detected synchronously, possibly, with integrationacross several frames.

It was also disclosed that coupling such a scanning detection devicewith an imaging camera may be particularly advantageous, as thefluorescent image from the scanner may be superimposed upon the colorimage from the camera to provide geometrically-accurate,clinically-relevant information not otherwise visible to the surgeon.

To realize full benefits of such alignment, it is important to establishthe correspondence between data captured by the scanning device 201 andimaging camera 203, (FIG. 11). This is facilitated by projecting a frame2 around the field of view (FOV) of the scanning device 1, or some otherregistration elements which are fixed in the FOV, such as corners. Forbest results, the frame rate of the camera should be synchronized withthat of the scanner, via camera's trigger input or other arrangement.Assuming that the entire scanning FOV is contained within the camera FOV4, the position of such registration elements can be detected by thecamera and their coordinates within the camera FOV can be established.Then the coordinates of all other pixels of the scanning device can befound within the camera FOV by interpolation.

If the target surface is not planar, the registration elements may notbe able to convey all the information needed for alignment every pixelof both scanner's and camera's images. In this case the entire projectedimage may be analyzed and used for alignment.

However, inclusion of the projected registration elements, as well asdetected fluorescent structures 5, may degrade the quality of the cameraimage. To avoid that, a camera can capture frames with variable timingand the image processing software may process frames in two streams, asdepicted on FIG. 12. In this ease “bright” frames 226 are captured whileprojection frames 223 are active and used for alignment only, while“dark” frames 225 are captured while projection frames 223 are notactive and used for fusion with bio-fluorescence data. The exposure of“bright” and “dark” frames may be different. Additionally, partial“bright” frames may be captured during a single camera exposure andsubsequently stitched in software. This would have an advantage ofcapturing more “dark” frames and hence providing fused frames withclinically-relevant information at higher rate, while “bright” framescaptured at lower rate may still provide sufficient alignment precision.

Additionally, still referring to FIG. 12, non-active projection periods224, during which all lasers of the scanning device are off, can be usedto capture so-called “black” frames from the scanning device, i.e.frames which contains no fluorescence data, just noise. The data inthose frames may be filtered or otherwise processed, stored, and thensubtracted from frames with valid data. While thermal noise and someother forms of noise are non-repeatable and hence cannot be canceled outthis way, the ambient light noise and the electronic interference frominternal and external sources may me repeatable and hence may be reducedor eliminated by black frame subtraction.

The electronic alignment by registration elements as described above mayneed considerable processing resources. In some cases it may beadvantageous to align the scanning device and a cameraopto-mechanically, in such a way that their optical axes are co-locatedalong the same line 6 when reaching the target surface 8 (FIG. 13). Toachieve this, a coaxial coupling element 207 is employed. Such couplingelement may be a dichroic mirror (if the wavelengths used by thescanning device and the camera are sufficiently different), or apolarizing mirror or polarizing cube (if the light used by the scanningdevice is linearly polarized and the camera can tolerate the loss ofhalf the light), or even a half-mirror (if both the scanning device andthe camera can tolerate the loss of some light). Other configurations ofthe coupling element are possible too.

If a coaxial coupling element is not feasible, a small coupling mirror227 placed right outside of the camera FOV may be employed to bring theFOVs of the scanning device and the camera to nearly-coaxial direction(FIG. 14). In this case, some electronic alignment may still benecessary, however, the computational intensity and the precision ofsuch alignment are greatly improved.

If mirror 227 is significantly smaller than the camera's aperture, itmay be employed within the camera FOV, as per FIG. 15. In this case, itblocks some of the aperture, but the amount of light entering the cameraaround it may still be sufficient.

It may also be advantageous to employ an additional objective 211, whichwould create a virtual image of the origin point of the scannersomewhere near the mirror 227, thus reducing the required size of themirror 227. Similar optical arrangement with an additional objective maybe used for the camera as well.

No matter which arrangement is used for the coupling, it is advantageousto co-locate the origin points of the scanning device and the camera, sothe relative size of their FOVs stays constant or nearly constant,irrespective of the distance to the target.

While a laser scanning device is capable of re-projecting the collectedbio-luminescent information onto the target, it may be advantageous touse a different, non-scanning projector for this purpose. The advantagesof non-scanning projectors may include higher light output and lowercost. It is conceivable to use a powerful DLP or LCoS-based non-scanningprojector as a replacement of a surgical light, so the projected imagewill not have to compete with ambient light.

As with cameras, for best results, the frame rate of a projector shouldbe synchronized with that of the scanner.

All of the above-mentioned alignment methods can be used for an imagingprojector as well. This is illustrated by an example on FIG. 16, wheretwo coupling mirrors 27 a and 27 b are placed within the FOV of aprojector 10 (most imaging projectors have fairly large apertures).Additional objectives 211 a and 211 b insure the smallest possible sizeof coupling mirrors, and hence, low loss of projector's light. Aparabolic hot mirror 209 is also shown, collecting the fluorescent lightinto a detector 212. This arrangement assumes that the fluorescent lighthas longer wavelength than visible light (emitted by the projector andcaptured by the camera). Generally, a detector 212 may be collocatedwith the scanner 201, or be positioned in a different place, as theposition of the detector has little impact on device's sensitivity.

The projected light may hit the target surface in such a way that anabnormally large (FIG. 17) or abnormally small amount of light will bereflected toward the User's eyes, due to specular reflection. Thisproblem may be alleviated by a sensor wearable by the User near his/hereyes, which would provide feedback for the projector controller, andthus adjust the amount of light going to each particular pixel of theimage according to surface reflectance at that point in the direction ofthe User.

The visibility of the projected pattern 214 (FIG. 18), indicatingdetected fluorescence, may be enhanced, if it is divided into two ormore sub-areas, which blink in a synchronized fashion. Left part of FIG.18 shows a magnified projected pattern 214, which is on the right, where215 and 216 represent two such sub-areas, designated by differenthatching: for example, when areas 215 are lit up, areas 216 remain dark,and vice versa. Areas might be one pixel each, or larger.

A unique property of a scanning bio-fluorescence detection device is itsability to provide time-resolved data. To take advantage of it, theexcitation laser should emit pulses 217, their duration beingconsiderably shorter than the duration of a scanning pixel (FIG. 19).The detection system should also be fast enough to be able to takemultiple read-outs 219 within a scanning pixel. Then, a temporalresponse 218 can be measured. This temporal data can be used to assessthe fluorophore temporal response, as in Fluorescence-Lifetime ImagingMicroscopy (FLIM), or to assess the time of light propagation from thefluorescence source, and hence, the depth of such source, enablingtomographic imaging.

For time-resolved measurements, it is especially advantageous to use asingle-photon counting detector. Then, instead of continuous responsecurve 218 as on FIG. 19, a number of pulses 220 would be detected, asdepicted on FIG. 20. Statistical analysis of their time of arrival canprovide the most accurate information about fluorescence-lifetime and/orthe depth of fluorescent sources in the body.

It may also be possible to use multiple excitation lasers emitting shortpulses within the same pixels and using the same detector to readmultiple fluorophores. In this case, preferably, the fluorophores shouldhave fairly short life time.

Additionally, the reflected light from one or more excitation lasers canbe detected and co-processed with fluorescent light, thus enhancing thecollected image.

It is hence advantageous to use fast, highly sensitive detectors, suchas Avalanche Photo-Diode (APD), Silicon Photo-Multiplier (SiPM),Photo-Multiplier Tube (PMT) Hybrid Photo-Multiplier (HPM) or other,characterized by short response time, high gain and low noise. It isalso advantageous to use detectors with large active area, as those maycollect more photons. Additionally, the size of optical collection area222 may grow proportionally to the active area of the detector 221, sothat

${d \approx {1\mspace{14mu}\ldots\mspace{14mu} 2*\frac{h*s}{A}}},$where d is the size of the optical collection area, S is the size of theactive area of the detector, A is the size of the target, and h is thedistance from the device to the target, as per FIG. 21.

Additionally, it may also be advantageous to employ multiple detectors221 a . . . 221 e, each with its own optical collection area 222 a . . .222 c, looking at the same target area (FIG. 22).

After both fluorescent image and color image are captured and aligned,various image fusion methods can be employed.

It may be particularly advantageous to capture the image formed byreflected excitation light in addition to the fluorescent image. Thereflected image is usually providing more detailed, higher resolutioninformation about the location of the fluorescent inclusions, while alsobeing perfectly aligned with fluorescent image. The image data from thecamera can then be used to colorize the reflected image, which otherwiseis black-and-white.

It may also be advantageous to co-process the fluorescent and reflectedimage, for example, normalizing the fluorescent data by reflected data.

Also, additional information may be extracted from the analysis oftemporal changes of the fluorescent images such as the direction andvelocity of the blood flow, the pulse or other physiological factors.The contrast and quality of the fluorescent image can also be improvedby storing the image taken before the fluorophore has been injected andcomparing it with the image after the fluorophore injection.

This invention claims:
 1. A method for three-dimensional imaging ofcancer cells of a target surgical area, and for projecting the imagedcancer cells directly onto the target surgical area, said methodcomprising: introducing, fluorophores having affinity for targetedcancer cells into biologic tissues of the target surgical area; emittinga beam of light at a first infrared wavelength from a first laser, abeam of light at a red wavelength from a second laser, a beam of lightat a blue wavelength from a third laser, and a beam of light at a greenwavelength from a fourth laser, within each of a plurality of alternateimaging frames; co-axially aligning the beams of infrared, red, blue,and green light using a means for aligning; scanning the co-axiallyaligned beam of light, using a scanner, in a pattern, and across thetarget surgical area, exciting the fluorophores and causing emitting offluorescent excitation light at a second infrared wavelength during thealternate imaging frames, and causing illumination of the targetsurgical area with white light during the alternate imaging frames;converting each image of the fluorescent excitation light of thefluorophores for each of the alternate imaging frames into an analogsignal by a light detector; converting the analog signal of each imageinto a digital image by an image processor, and successively storingeach in a memory; successively outputting each digital image of thestored image frames to one of said red, green, or blue lasers as ananalog signal by the processing unit, and projecting by said one of saidred, green, or blue lasers of each image onto the target surgical areaduring respective display frames succeeding the alternate imagingframes, using said analog signal.
 2. The method according to claim 1further comprising: selectively emitting a beam of ablation light, by afifth laser, at a selective wavelength and ablating of the targetedcancer cells; aligning the beam of ablation light with the co-axiallyaligned beam of light; and controlling said selectively emitted ablationlight by the fifth laser by the image processor for only occurring whendirected at the targeted cancer cells.
 3. The method according to claim1 further comprising: transmitting, by the image processor, each storeddigital image of the alternate imaging frames to a monitor forsuccessively displaying of each image thereon.
 4. The method accordingto claim 1 further comprising: capturing a combined image of the targetsurgical area and the projection of each image on the target surgicalarea using a camera, and displaying said captured image on a monitor. 5.The method according to claim 4 further comprising synchronizing a framerate of the camera with a frame rate of the scanner.
 6. The methodaccording to claim 1 wherein the first infrared wavelength of light isat a wavelength of 780 nm.
 7. The method according to claim 6 whereinthe red wavelength of light is at a wavelength of 640 nm.
 8. The methodaccording to claim 7 wherein said second infrared wavelength of light isapproximately at a wavelength of 820 nm.
 9. The method according toclaim 8 wherein the blue wavelength of light is at a wavelength of 440nm.
 10. The method according to claim 9 wherein the green wavelength oflight is in the range of wavelengths between 510 nm and 540 nm.